1. Field of the Invention
The present invention relates to reduction of motion compensation artifacts in an image interpolation more specifically to halo reduction in interpolated images.
2. Discussion of Related Art
Image interpolation based on motion compensation is a well-established technique. One example of motion compensation is frame-rate conversion (FRC), which is often utilized to increase (or decrease) the refresh rate in video and to have more images (or fewer images) shown per second. One advantage of interpolating video to a higher number of images per second is that, with the higher refresh rates available in LCD panels, motion appears more fluid. Another advantage is in the conversion of film content, which is normally at 24 frames per second (fps), to video, which could be 30, 50, 60, 100, 120, 200 or 240 fps. An advantage of interpolating to fewer images is in conversion between commonly used standards such as 60 Hz for National Television System Committee (NTSC) to 50 fps for the Phase Altering Line (PAL) standard.
FRC is very challenging, particularly in real-time video. Many previous solutions have attempted to address this issue; however, these solutions all introduce artifacts, especially in real-time applications. Most prior art methods use block-based motion vector (MV) estimation for motion compensation (MC) and motion estimation (ME). These prior art solutions, even for progressive images, cause many artifacts, including the halo effect, flickering, and block artifacts.
Block artifacts can occur when the MV estimation is based on a block of pixels of fixed and predetermined size. In this case, all pixels in a block are supposed to have the same MV even if within a block there are different objects or regions moving in different manners. Moreover, when adjacent blocks to a given block have different estimated motion vectors for any reason, the block boundaries become more obvious as same-region pixels will be displaced differently in different adjacent blocks.
Flickering is another artifact in motion compensated image interpolation. This artifact occurs when the motion vectors of a few pixels in the image sequence are not consistent from one frame to another frame. The inconsistency is generally due to erroneous motion vector calculations in a small region.
Halo effects occur around a foreground object in an image in a video sequence when the object is moving relative to a background. The background could be stationary and the object moving or vice versa. In some cases, both the object and the background are moving, as may be the case, for example, in a video game sequence. The origin of the halo effect is erroneous MV estimation in occlusion areas. The visual perception of the halo effect is that objects seem to have a halo of different characteristic around them. For example, the face of a person walking in a scene could seem to be surrounded by a halo, as if a portion of the background was moving along with the face.
There have been various halo reduction algorithms developed. Halo reduction can be based on Object-based Interpolation as described in U.S. Pat. No. 6,625,333. However, this technique implies object segmentation and object coloring, which requires additional frame memory and additional bandwidth. Moreover, the proposed MV correction requires multiple frame-based iterations which are not suitable for real-time processing.
Multi-frame solutions, those that involve analysis of more than two frames of data, have also been proposed. See, e.g., U.S. Pat. No. 6,005,639; U.S. Pat. No. 6,011,596; U.S. Pat. No. 7,010,039; U.S. Application Publication No. 2007/0121725A1; and K. Sugiyama, T. Aoki & S. Hangai, “Motion Compensated Frame Rate Conversion Using Normalized Motion Estimation”, 2005 IEEE Workshop on Signal Processing Systems—Design and Implementation (SiPS 2005), Athens, Greece. Multi frame solutions for covering/uncovering/no occlusion detection and MV correction are based essentially from previous-previous and far-future images frames. Multi-frame solutions require large image memory and highly increased bandwidth between the memory and the processor.
Two-frame solutions that have been proposed are based on MV correction (or re-estimation), Interpolation strategies, and/or a mixing of these two methods. In MV correction, separating foreground and background motion vectors is also a possible way to reduce the halo effects. The use of some motion vectors set at borders of the image as possible background MV has been proposed in U.S. Publication No. US 2006/0072790 A1. The assumption that the border MV is the background MV, however, is not necessarily valid and the background MV estimation requires additional frame latency and extra memory.
Median Filtering, an example of interpolation strategy, is proposed in G. de Haan, “IC for Motion-Compensated De-Interlacing, Noise Reduction and Picture Rate Conversion”, IEEE Trans. on CE, Vol. 45, No. 3, August 1999. In this reference, Median Filtering Interpolation is the median value of forward image interpolation, backward image interpolation, and the averaging value of the two existing images. Moreover, forward and backward MV are obtained using a single recursively estimated MV. Median Filtering Interpolation, as described, can yield generally good results in no-occlusion regions. However, in occlusion regions the estimated MV utilizing the Median Filtering method is no longer correct.
A weighted mean of individual interpolations is proposed in U.S. Pat. No. 7,039,109. Each individual interpolation provided by the MV in a neighbor group is based again on Median Filtering of forward and backward interpolations and the mean value of the two existing images. MVs described in this patent are block-based and forward estimated from the present image to the next image. The weight specified for an individual interpolation is a measure of reliability in function of the difference of a predicted luminous intensity and/or of the relative frequency of occurrence of the MV in the neighboring groups. MV edge detection is also provided for complexity reduction purposes. However, the group or block-based MV solution can provide blocking effects. Additionally, a single (forward) MV for a block can be insufficient in occlusion parts of the picture. Further, MV edge detection is only good for software simulation. Also, pixel-based image averaging can blur the interpolated image in the occlusion regions or yield an image doubling effect when the MV is large.
Covering/Uncovering Detection, a second block-based ME with error at 2nd or 4th power, is proposed for more precision in U.S. Pat. No. 6,219,436. MV Correction by using MV at shifted locations in the function of detected “real” covering uncovering regions is proposed. However, various median filtering strategies with appropriate mean values for detected regions are in fact the preferable solution presented in the patent.
In both U.S. Pat. No. 6,487,313 and U.S. Pat. No. 7,058,227, it is assumed that the discontinuities in MV field correspond to the borders of moving objects. The use of MV length to determine the occlusion areas, an additional bidirectional interpolation error calculation with small block size for increasing the precision of detected occlusion areas, and various median filtering for final interpolation is thus proposed. However, MV length is not necessarily a reliable parameter when the objects are composed of flat but noisy regions. The discontinuities in MV field and borders of moving objects require some expensive precision, and median filtering is still an ad-hoc technique since MV is not corrected.
Therefore, there is a need to provide a fast interpolation of image planes that does not require large amounts of memory to accomplish.